{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autoscaling Seldon Deployments\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "You will need\n",
    "\n",
    " - [Git clone of Seldon Core](https://github.com/SeldonIO/seldon-core)\n",
    " - A running Kubernetes cluster with kubectl authenticated\n",
    " \n",
    "    - The cluster should have `heapster` and `metric-server` running in the `kube-system` namespace\n",
    "    - For Minikube run:\n",
    "    \n",
    "    ```\n",
    "    minikube addons enable metrics-server\n",
    "    minikube addons enable heapster\n",
    "    ```\n",
    "    \n",
    " - [seldon-core Python package](https://pypi.org/project/seldon-core/) (```pip install seldon-core```)\n",
    " - [Helm client](https://helm.sh/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Kubernetes Cluster\n",
    "\n",
    "Follow the [Kubernetes documentation to create a cluster](https://kubernetes.io/docs/setup/).\n",
    "\n",
    "Once created ensure ```kubectl``` is authenticated against the running cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"gke_seldon-demos_europe-west4-c_standard-cluster-1\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/tiller created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/tiller created\n",
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"tiller-deploy\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Thu Aug 29 13:09:08 2019\n",
      "NAMESPACE: seldon-system\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ClusterRole\n",
      "NAME                          AGE\n",
      "seldon-operator-manager-role  2s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME                                 AGE\n",
      "seldon-operator-manager-rolebinding  1s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-config            1     2s\n",
      "seldon-spartakus-config  1     2s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                         READY  STATUS             RESTARTS  AGE\n",
      "seldon-operator-controller-manager-0         0/1    ContainerCreating  0         1s\n",
      "seldon-spartakus-volunteer-5b568c587b-ww66l  0/1    ContainerCreating  0         1s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                                   TYPE    DATA  AGE\n",
      "seldon-operator-webhook-server-secret  Opaque  0     2s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                                        TYPE       CLUSTER-IP   EXTERNAL-IP  PORT(S)  AGE\n",
      "seldon-operator-controller-manager-service  ClusterIP  10.0.22.99   <none>       443/TCP  1s\n",
      "webhook-server-service                      ClusterIP  10.0.20.211  <none>       443/TCP  1s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                              SECRETS  AGE\n",
      "seldon-core-seldon-core-operator  1        2s\n",
      "seldon-spartakus-volunteer        1        2s\n",
      "\n",
      "==> v1/StatefulSet\n",
      "NAME                                READY  AGE\n",
      "seldon-operator-controller-manager  0/1    1s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  2s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  1s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  2s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                        READY  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  0/1    1           0          1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for 1 pods to be ready...\n",
      "partitioned roll out complete: 1 new pods have been updated...\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Ingress\n",
    "Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   ambassador\n",
      "LAST DEPLOYED: Thu Aug 29 13:10:03 2019\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Deployment\n",
      "NAME        READY  UP-TO-DATE  AVAILABLE  AGE\n",
      "ambassador  0/3    3           0          1s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                         READY  STATUS             RESTARTS  AGE\n",
      "ambassador-684d6f8cd9-cfxwc  0/1    ContainerCreating  0         1s\n",
      "ambassador-684d6f8cd9-lxwcd  0/1    ContainerCreating  0         1s\n",
      "ambassador-684d6f8cd9-ncv8b  0/1    ContainerCreating  0         1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME              TYPE          CLUSTER-IP   EXTERNAL-IP  PORT(S)                     AGE\n",
      "ambassador        LoadBalancer  10.0.21.195  <pending>    80:30644/TCP,443:32376/TCP  1s\n",
      "ambassador-admin  ClusterIP     10.0.30.220  <none>       8877/TCP                    1s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME        SECRETS  AGE\n",
      "ambassador  1        1s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME             AGE\n",
      "ambassador       1s\n",
      "ambassador-crds  1s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME             AGE\n",
      "ambassador       1s\n",
      "ambassador-crds  1s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                          AGE\n",
      "authservices.getambassador.io                 1s\n",
      "consulresolvers.getambassador.io              1s\n",
      "kubernetesendpointresolvers.getambassador.io  1s\n",
      "kubernetesserviceresolvers.getambassador.io   1s\n",
      "mappings.getambassador.io                     1s\n",
      "modules.getambassador.io                      1s\n",
      "ratelimitservices.getambassador.io            1s\n",
      "tcpmappings.getambassador.io                  1s\n",
      "tlscontexts.getambassador.io                  1s\n",
      "tracingservices.getambassador.io              1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Congratuations! You've successfully installed Ambassador.\n",
      "\n",
      "For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.\n",
      "\n",
      "To get the IP address of Ambassador, run the following commands:\n",
      "NOTE: It may take a few minutes for the LoadBalancer IP to be available.\n",
      "     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'\n",
      "\n",
      "  On GKE/Azure:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\n",
      "\n",
      "  On AWS:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')\n",
      "\n",
      "  echo http://$SERVICE_IP:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install stable/ambassador --name ambassador --set crds.keep=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"ambassador\" rollout to finish: 0 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 1 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 2 of 3 updated replicas are available...\n",
      "deployment \"ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deployment.apps/ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create model with autoscaler\n",
    "\n",
    "To create a model with an HorizontalPodAutoscaler there are three steps:\n",
    "\n",
    "\n",
    "  1. Ensure you have a resource request for the metric you want to scale on if it is a standard metric such as cpu or memory, e.g.:\n",
    "  \n",
    "```\n",
    "      \"resources\": {\n",
    "         \"requests\": {\n",
    "            \"cpu\": \"0.5\"\n",
    "         }\n",
    "      }\n",
    "```\n",
    "     \n",
    "  1. Add an HPA Spec refering to this Deployment, e.g.:\n",
    "  \n",
    "```\n",
    "     \"hpaSpec\":\n",
    "\t\t    {\n",
    "\t\t\t\"minReplicas\": 1,\n",
    "\t\t\t\"maxReplicas\": 3,\n",
    "\t\t\t\"metrics\": \n",
    "\t\t\t    [ {\n",
    "\t\t\t\t\"type\": \"Resource\",\n",
    "\t\t\t\t\"resource\": {\n",
    "\t\t\t\t    \"name\": \"cpu\",\n",
    "\t\t\t\t    \"targetAverageUtilization\": 10\n",
    "\t\t\t\t}\n",
    "\t\t\t    }]\n",
    "\t\t    },\n",
    "```\n",
    "\n",
    "The full SeldonDeployment spec is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-model\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"cpu\"\u001b[39;49;00m: \u001b[33m\"0.5\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m\r\n",
      "                    },\r\n",
      "\t\t    \u001b[34;01m\"hpaSpec\"\u001b[39;49;00m:\r\n",
      "\t\t    {\r\n",
      "\t\t\t\u001b[34;01m\"minReplicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\t\u001b[34;01m\"maxReplicas\"\u001b[39;49;00m: \u001b[34m3\u001b[39;49;00m,\r\n",
      "\t\t\t\u001b[34;01m\"metrics\"\u001b[39;49;00m: \r\n",
      "\t\t\t    [ {\r\n",
      "\t\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"Resource\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\u001b[34;01m\"resource\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"cpu\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t    \u001b[34;01m\"targetAverageUtilization\"\u001b[39;49;00m: \u001b[34m10\u001b[39;49;00m\r\n",
      "\t\t\t\t}\r\n",
      "\t\t\t    }]\r\n",
      "\t\t    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"example\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize model_with_hpa.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-model created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f model_with_hpa.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node/gke-standard-cluster-1-default-pool-b1c35e14-rrbd labeled\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   loadtest\n",
      "LAST DEPLOYED: Thu Aug 29 13:17:11 2019\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Pod(related)\n",
      "NAME                   READY  STATUS             RESTARTS  AGE\n",
      "locust-master-1-znncw  0/1    ContainerCreating  0         0s\n",
      "locust-slave-1-hnx8n   0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ReplicationController\n",
      "NAME             DESIRED  CURRENT  READY  AGE\n",
      "locust-master-1  1        1        0      0s\n",
      "locust-slave-1   1        1        0      0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME             TYPE      CLUSTER-IP   EXTERNAL-IP  PORT(S)                                       AGE\n",
      "locust-master-1  NodePort  10.0.31.100  <none>       5557:32552/TCP,5558:32023/TCP,8089:32677/TCP  0s\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-loadtesting --name loadtest  \\\n",
    "    --set locust.host=http://seldon-model-test-deployment-example:8000 \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=1 \\\n",
    "    --set loadtest.sendFeedback=0 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a few mins you should see the deployment `my-dep` scaled to 3 deployments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                                   READY   STATUS    RESTARTS   AGE\r\n",
      "pod/ambassador-684d6f8cd9-cfxwc                        1/1     Running   0          10m\r\n",
      "pod/ambassador-684d6f8cd9-lxwcd                        1/1     Running   0          10m\r\n",
      "pod/ambassador-684d6f8cd9-ncv8b                        1/1     Running   0          10m\r\n",
      "pod/locust-master-1-znncw                              1/1     Running   0          3m13s\r\n",
      "pod/locust-slave-1-hnx8n                               1/1     Running   0          3m13s\r\n",
      "pod/test-deployment-example-7cd068f-6cc64774ff-dtqwv   2/2     Running   0          2m40s\r\n",
      "pod/test-deployment-example-7cd068f-6cc64774ff-gdkb8   2/2     Running   0          8m57s\r\n",
      "pod/test-deployment-example-7cd068f-6cc64774ff-l4mn5   2/2     Running   0          5m11s\r\n",
      "\r\n",
      "NAME                                                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\n",
      "deployment.extensions/ambassador                        3         3         3            3           10m\r\n",
      "deployment.extensions/test-deployment-example-7cd068f   3         3         3            3           8m57s\r\n",
      "\r\n",
      "NAME                                                                  REFERENCE                                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE\r\n",
      "horizontalpodautoscaler.autoscaling/test-deployment-example-7cd068f   Deployment/test-deployment-example-7cd068f   51%/10%   1         3         3          8m57s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pods,deployments,hpa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove Load\n",
    "After 5-10 mins you should see the deployments replicas decrease to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "release \"loadtest\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!helm delete loadtest --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                                   READY   STATUS    RESTARTS   AGE\r\n",
      "pod/ambassador-684d6f8cd9-cfxwc                        1/1     Running   0          16m\r\n",
      "pod/ambassador-684d6f8cd9-lxwcd                        1/1     Running   0          16m\r\n",
      "pod/ambassador-684d6f8cd9-ncv8b                        1/1     Running   0          16m\r\n",
      "pod/test-deployment-example-7cd068f-6cc64774ff-gdkb8   2/2     Running   0          15m\r\n",
      "\r\n",
      "NAME                                                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\n",
      "deployment.extensions/ambassador                        3         3         3            3           16m\r\n",
      "deployment.extensions/test-deployment-example-7cd068f   1         1         1            1           15m\r\n",
      "\r\n",
      "NAME                                                                  REFERENCE                                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE\r\n",
      "horizontalpodautoscaler.autoscaling/test-deployment-example-7cd068f   Deployment/test-deployment-example-7cd068f   1%/10%    1         3         1          15m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pods,deployments,hpa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
